{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spam Classification with NLP in Python\n",
    "\n",
    "In this project, we classify a messages as Spam or Ham (not spam), from a dataset of 5000+ SMS phone messages, available from [UCI]((https://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection)). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file is a tab separated txt file, containing the label of a message, and the message itself.\n",
    "\n",
    "Let's use Pandas' read_csv method, and a custom separator to read this in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                            message\n",
       "0   ham  Go until jurong point, crazy.. Available only ...\n",
       "1   ham                      Ok lar... Joking wif u oni...\n",
       "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3   ham  U dun say so early hor... U c already then say...\n",
       "4   ham  Nah I don't think he goes to usf, he lives aro..."
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages = pd.read_csv('data/SMSSpamCollection', sep='\\t',\n",
    "                           names=[\"label\", \"message\"])\n",
    "messages.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory Data Analysis\n",
    "\n",
    "Let's check out some of the stats with some plots and the built-in methods in pandas!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>5572</td>\n",
       "      <td>5572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>2</td>\n",
       "      <td>5169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>ham</td>\n",
       "      <td>Sorry, I'll call later</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>4825</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       label                 message\n",
       "count   5572                    5572\n",
       "unique     2                    5169\n",
       "top      ham  Sorry, I'll call later\n",
       "freq    4825                      30"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grouping by label, we can begin to think about the features that separate ham and spam!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>message</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>label</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">ham</th>\n",
       "      <th>count</th>\n",
       "      <td>4825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>4516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Sorry, I'll call later</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">spam</th>\n",
       "      <th>count</th>\n",
       "      <td>747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Please call our customer service representativ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                        message\n",
       "label                                                          \n",
       "ham   count                                                4825\n",
       "      unique                                               4516\n",
       "      top                                Sorry, I'll call later\n",
       "      freq                                                   30\n",
       "spam  count                                                 747\n",
       "      unique                                                653\n",
       "      top     Please call our customer service representativ...\n",
       "      freq                                                    4"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages.groupby('label').describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a new column to detect how long the text messages are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>message</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "      <td>61</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                            message  length\n",
       "0   ham  Go until jurong point, crazy.. Available only ...     111\n",
       "1   ham                      Ok lar... Joking wif u oni...      29\n",
       "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...     155\n",
       "3   ham  U dun say so early hor... U c already then say...      49\n",
       "4   ham  Nah I don't think he goes to usf, he lives aro...      61"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages['length'] = messages['message'].apply(len)\n",
    "messages.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x114eb8e80>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAFoCAYAAABNMRFVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xu4XXV95/F3cmIOCeaYGKeGNtCYQb9cSmm4FNCIculU\nn1ZE7FNvdQbQSEcZB5gMjFXBylg1RUAcKYogD2OHcZgatFCliB0DlUi4lNIAXyl3SIwmBk8MuZiT\nzB9rbd2enOvO3uvsdfJ+PU8ezl7ftfb+7W/OOXzyW7+19pRdu3YhSZJUR1MnegCSJEmtMshIkqTa\nMshIkqTaMshIkqTaMshIkqTaMshIkqTaMshIkqTaMshIkqTaMshIkqTamjbRA2gWEb3APcAHMnNF\nuW1/4AvA64DngA9n5o1Nx5wMXAYsBO4ClmTmE031c4ClwCzgRuDszNxazTuSJEmd1DUzMmWIuQE4\npGlbD/B3wFbgd4BLgK9ExCFlfX9gOXANcBSwHrip6fi3AhcCS4ATgWOBZRW8HUmSVIGuCDIRcTCw\nEnjFoNIfAL8BvDszH83MLwK3AK8u6+8FVmXm5Zn5MHAGsCAiji/rHwQuy8xvZua9wFnAeyJinw6/\nJUmSVIGuCDIUp41uB44DpgzenpmbGxsy87TM/FL58FhgRVNtC3AfcFxETAWOBu5oer6VwHTg8E68\nCUmSVK2uWCOTmVc1vo6I5tJC4ImI+CTwbuDHwMcy8+tlfT9gzaCnWwfMB2YD+zTXM3MgIjaU9e+3\n+W1IkqSKdcuMzHBeTHG6aDbwh8D/BP5vRBxR1mcC2wYdsw3oLWuMUJckSTXXFTMyI9gBrM/M/1g+\n/qeIeC3wPuBPKRYBDw4lvcDGssYw9RfGOoBdu3btmjJlyug7SpKkwTr+P9BuDzJrgZ2DtiVwWPn1\nc8C8QfV5wP3ABoowMw/4AfziKqi55fOOyZQpU+jv38LAwOBhqBN6eqbS1zfDnlfInlfPnlfPnlev\n0fNO6/YgsxL4cERMycxd5baDgSeb6osbO0fETGARcGFm7oqIVWW9sSD41cB24IHxDGJgYCc7dviN\nXyV7Xj17Xj17Xj17Pvl0e5C5AfgocGVEXAL8PvAG4HfL+rXA0og4H7gZuAh4vHEzPeBK4KqIWE2x\n6PdK4IveEE+SpMmhGxf7NmZeyMxNwO9RzMI8CPwn4I8z84Gy/hRwGnAmcDfFouBTm47/KvBJijsD\n30px598LKnkXkiSp46bs2rVr9L32brs2btzsVGRFpk2bypw5+2LPq2PPq2fPq2fPq1f2vOOLfbtx\nRkaSJGlMDDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKS\nJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2DDKSJKm2\nDDKSJKm2DDKSJKm2pk30AOpu+/btrF794Ij7HHroYUyfPr2iEUmStPcwyOyh1asf5PxLv8asuQcM\nWd+04WmWnQeLFh1Z8cgkSZr8DDJtMGvuAcye98qJHoYkSXsd18hIkqTaMshIkqTaMshIkqTa6qo1\nMhHRC9wDfCAzVwyq9QEPAX+Wmdc3bT8ZuAxYCNwFLMnMJ5rq5wBLgVnAjcDZmbm10+9FkiR1XtfM\nyJQh5gbgkGF2WQbsN+iY/YHlwDXAUcB64Kam+luBC4ElwInAseXzSJKkSaArgkxEHAysBF4xTH0x\nRRD54aDSe4FVmXl5Zj4MnAEsiIjjy/oHgcsy85uZeS9wFvCeiNinE+9DkiRVqyuCDPA64HbgOGBK\ncyEipgNfBN4PbB903LHAL05BZeYW4D7guIiYChwN3NG0/0pgOnB4m8cvSZImQFcEmcy8KjOXDrN2\n5cPAvZn57SFq+wFrBm1bB8wHZgP7NNczcwDYUNYlSVLNddVi38Ei4hDgfcBhw+wyE9g2aNs2oLes\nMUJ9zHp6hs97I9Wa95k2rSsyY9dr9HMsfVV72PPq2fPq2fPqVdXrrg4yFKeULszM9cPUt7J7KOkF\nNpY1hqm/MJ5B9PXNaKnWvM+cOfuO5yX3emPpq9rLnlfPnlfPnk8+XRtkIuIA4NXAb0fEpeXmmcAX\nIuJtmfkHwHPAvEGHzgPupziFtLV8/IPyOXuAucDa8Yylv38LAwM7h62N5fiNGzeP5yX3Wj09U+nr\nmzFiz9Ve9rx69rx69rx6jZ53WtcGGeBZ4MBB274LfBb46/LxSmBxoxgRM4FFFLM4uyJiVVlvLAh+\nNcWC4QfGM5CBgZ3s2DH0N/5YfiBGOl5Ds2fVs+fVs+fVs+eTT9cGmczcCTzevC0idgA/yszGjMq1\nwNKIOB+4GbgIeLzpZnpXAldFxGqKRb9XAl/0hniSJE0O3bjqaddYa5n5FHAacCZwN8WVSqc21b8K\nfBL4AnArxZ1/L2jzeCVJ0gTpuhmZzOwZobZwiG23AgeNcMwyvJuvJEmTUjfOyEiSJI2JQUaSJNWW\nQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaS\nJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWW\nQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNXWtIkeQLOI6AXuAT6QmSvKbccCnwF+G3gWuCQz\nr2k65mTgMmAhcBewJDOfaKqfAywFZgE3Amdn5tZq3pEkSeqkrpmRKUPMDcAhTdteDvwd8B3gd4CP\nAZ+LiDeW9QOA5cA1wFHAeuCmpuPfClwILAFOBI4FlnX+3UiSpCp0RZCJiIOBlcArBpVOBdZm5kcz\n87HM/CpwPfDOsv5eYFVmXp6ZDwNnAAsi4viy/kHgssz8ZmbeC5wFvCci9un0e5IkSZ3XFUEGeB1w\nO3AcMKVp+zcpwslgLyn/ewyworExM7cA9wHHRcRU4GjgjqbjVgLTgcPbNnJJkjRhumKNTGZe1fg6\nIpq3Pw083VT7NeDtFKeLAPYD1gx6unXAfGA2sE9zPTMHImJDWf9+W9+EJEmqXLfMyIyqPB30NxTB\n5Ivl5pnAtkG7bgN6yxoj1CVJUs11xYzMaCJiX+AbwIHAa5quOtrK7qGkF9hY1him/sJ4Xr+nZ/i8\nN1KteZ9p02qTGSdUo59j6avaw55Xz55Xz55Xr6ped32QiYhZwLcoLq8+ITMfbyo/B8wbdMg84H5g\nA0WYmQf8oHyuHmAusHY8Y+jrm9FSrXmfOXP2Hc9L7vXG0le1lz2vnj2vnj2ffLo6yETEFIrLqxcA\nx2fmo4N2WQksbtp/JrAIuDAzd0XEqrLeWBD8amA78MB4xtHfv4WBgZ3D1sZy/MaNm8fzknutnp6p\n9PXNGLHnai97Xj17Xj17Xr1Gzzutq4MMxeXVrwfeBPSX95UB2J6ZG4FrgaURcT5wM3AR8HjjZnrA\nlcBVEbGaYm3NlcAXx3tDvIGBnezYMfQ3/lh+IEY6XkOzZ9Wz59Wz59Wz55NPN54s3FX+ATiN4nLs\nmymCSOPP3wBk5lPlPmcCd1NcqXRq44nK+858EvgCcCvFnX8vqOJNSJKkzuu6GZnM7Gn6+o1j2P9W\n4KAR6svwbr6SJE1K3TgjI0mSNCYGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsG\nGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmS\nVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsGGUmSVFsG\nGUmSVFvTJnoAzSKiF7gH+EBmrii3LQCuBo4DngTOzczbmo45GbgMWAjcBSzJzCea6ucAS4FZwI3A\n2Zm5tYr3I0mSOqtrZmTKEHMDcMig0k3AGuBI4CvA8oiYXx6zP7AcuAY4Clhf7t94zrcCFwJLgBOB\nY4FlHX0jkiSpMl0RZCLiYGAl8IpB20+kmGk5Kwufoph1ObPcZQmwKjMvz8yHgTOABRFxfFn/IHBZ\nZn4zM+8FzgLeExH7dP5dSZKkTuuKIAO8Drid4vTRlKbtxwD3DToVdGe5X6O+olHIzC3AfcBxETEV\nOBq4o+nYlcB04PB2vwFJklS9rlgjk5lXNb6OiObSfhSnlZqtA+aPoT4b2Ke5npkDEbGhrH+/HWOX\nJEkTp1tmZIYzE9g2aNs2oHcM9ZlNj4c7XpIk1VhXzMiMYCvw0kHbeoEXmuqDQ0kvsLGsMUz9Bcah\np2f4vDdSDWDnwA4efTRH3e+3fuswpk+fPp5hTUqNPo3WL7WPPa+ePa+ePa9eVb3u9iDzHLtfxTQP\nWNtUnzdE/X5gA0WYmQf8ACAieoC5TcePSV/fjJZqAJufX8vVf7uGWd/bNOw+mzY8zdUXz+Doo48e\nz7AmtdH6qvaz59Wz59Wz55NPtweZlcAFEdGbmY1TRIv55QLeleVjACJiJrAIuDAzd0XEqrLeWBD8\namA78MB4BtHfv4WBgZ3D1kYza+4BzJ73ylFfY+PGzeMZ1qTU0zOVvr4ZI/Zc7WXPq2fPq2fPq9fo\nead1e5D5LvAMcF1EXAycQnEl0ull/VpgaUScD9wMXAQ83riZHnAlcFVErKZY9Hsl8MXx3hBvYGAn\nO3YM/Y3frh+IkV5jb2Q/qmfPq2fPq2fPJ59uPFm4q/FFZu4E3kxxeuge4J3AqZn5bFl/CjiN4r4y\nd1NcqXRq0/FfBT4JfAG4leIeNBdU8i4kSVLHdd2MTGb2DHr8OHDCCPvfChw0Qn0Z3s1XkqRJqRtn\nZCRJksakpRmZiPg+xfqU/52ZP23vkCRJksam1RmZ7wAfBtZGxA0R8e8iYspoB0mSJLVTS0EmMz8E\n/CbFQtwdwNeApyPiExHxqjaOT5IkaVgtr5HJzF2ZeVtmvhv4NeDzwDnAwxGxIiJOa9cgJUmShrJH\nVy1FxH7An5R/DgP+EbgO2B/4UkQcn5nn7OkgJUmShtLqYt8/Af49xWXRPwKuB/4oMx9t2udp4LMU\nszSSJElt1+qMzDUUd9I9FfhmeeO6wR4B/kerA5MkSRpNq0HmNyg+lPGljRATEb8L3JuZAwCZ+T3g\ne20ZpSRJ0hBaXez7EiD51dv93wI8EBH77/GoJEmSxqDVIHM58ChwadO2Q4CnB22TJEnqmFaDzGuB\n8zLzh40Nmflj4L8CJ7VjYJIkSaNpNcj8HJgzxPaZgHf4lSRJlWg1yHwTuCIi/m1jQ0QsBC4DvtWO\ngUmSJI2m1auWlgK3AT+IiI3ltjnAvcC57RiYJEnSaFoKMpn5o4g4AjgZ+C2KU00PAbdn5q42jk+S\nJGlYLX9EQXm/mFvLP5IkSZVr9SMK5gH/HXgNMJ1BC3wzc+GeD02SJGlkrc7IXA0cCfxv4KftG44k\nSdLYtRpkTgTekJl3tHMwkiRJ49Hq5dc/A9a1cyCSJEnj1WqQuR44PyJ62jkYSZKk8Wj11NLLgHcA\nfxgRjwHbmouZeeKeDkySJGk0LV9+DdzQtlFIkiS1oNUb4p3R7oFIkiSNV8szMhGxH7AEOAg4Bzge\neDAzs01jkyRJGlFLi30j4kDgX4DTgT8CXgy8DbgnIo5p2+gkSZJG0OqMzGeA5RQzMv3ltndQXM30\nKeCEPR9aISLmA39FMeOzAfhsZn62rC2guDnfccCTwLmZeVvTsSdTfCL3QuAuYElmPtGusUmSpInV\n6uXXrwEubf6AyMzcAXwcOKIdA2tyI7CpfN5zgE9ExJvL2teBNRR3Gf4KsLwMPkTE/hRh6xrgKGA9\ncFObxyZJkiZQq0GmZ5hj+4CB1ofzqyJiNnAM8N8z87HM/AbwLeCkiDgBeAVwVhY+RTHrcmZ5+BJg\nVWZenpkPA2cACyLi+HaNT5IkTaxWg8ytwIcionH8roh4KfBp4Pa2jKywBdgMnBER0yIiKGaD7geO\nBe7LzK1N+99JcZoJigC0olHIzC3AfU11SZJUc60GmfOAo4G1wAzgb4GnKNaiLG3P0CAztwFnA39K\nEWoeBv4uM78M7EdxWqnZOmB++fVodUmSVHOt3kdmTUT8DsUC30UUgehfgK9kZv+IB4/fwcA3gEuA\nw4DPRcTtwEwG3VG4fNxbfj1afcx6eobPeyPVxvsa06a157nqrNHPdvVVo7Pn1bPn1bPn1auq1y3f\nRyYzX6BYSNsxEXES8B5gfjk7c3+5mPcjFKew5g46pBd4ofx6K7uHll5g43jH0dc3o6XaeF9jzpx9\n2/Jck0G7+qqxs+fVs+fVs+eTT0tBJiK+M1K9jZ+1dATwaBliGu4H/gx4Djh00P7zKE53UdbnDVG/\nf7yD6O/fwsDAzmFr7dDfv4WNGze35bnqrKdnKn19M0bsudrLnlfPnlfPnlev0fNOa3VG5qkhnueV\nFKd+LtujEf2qNcCBETGtvLwbilNNTwArKRYc9zYFncXAHeXXK8vHAETETIrTYBeNdxADAzvZsWPo\nb/x2/UCM9Bp7I/tRPXtePXtePXs++bT1s5Yi4qPA/ns0ol/1t8Ay4EsR8QmKj0P4UPlnBfAMcF1E\nXAycQrEA+fTy2GuBpRFxPnAzRYB5LDO/28bxSZKkCdTulTj/E/jjdj1ZuXD4JIorkO6muKPwxzPz\nS5m5kyK8zAPuAd4JnJqZz5bHPgWcRnFfmbuB2cBb2jU2SZI08Vpe7DuMVwM7Rt1rHDLzEeD3h6k9\nzggfh5CZt1LM4kiSpEmonYt9+4DDgc/v0YgkSZLGqNUZmaeBXYO2bQf+B8VnHkmSJHVcq4t9T2/z\nOCRJksat1VNLY/7gxcxcMfpekiRJ49fqqaX/xy9PLU1p2j542y6KT8qWJElqu1Yvv34T8CTFpdb/\nhmKh70lAUtzj5RXln4V7PkRJkqShtTojcynwgcz8VtO2f4iIs4DrM3PZng9NkiRpZK3OyPwGu39M\nAUA/xQyNJElSx7UaZO4C/iIiZjU2RMRLKT5O4NvtGJgkSdJoWj219EHgH4DnIuIHFIHoVRSfPD3s\nnXYlSZLaqaUZmcx8mOJTqP8bxadMf48i3Bze+KwjSZKkTmv5s5Yyc2NEfIni6qTHy20/b9fAJEmS\nRtPqDfGmAJ+kmIWZTnFa6RMRsRn4jwYaSZJUhVYX+/4n4N3A+4Ft5babgLcAH9vzYUmSJI2u1SBz\nFnB2Zl4H7ATIzK8C7wXe1Z6hSZIkjazVIPMK4P4htj8AzGt9OJIkSWPXapB5Ejh6iO1vpFz4K0mS\n1GmtXrX0l8CVEbEfRRg6KSLeR7H497x2DU6SJGkkLQWZzPxyRLwI+AgwA/gC8GPgI5l5VRvHJ0mS\nNKxWL79+B3BjZn4xIl4GTM3MH7V3aJIkSSNr9dTS54HFwMbMXN/G8UiSJI1Zq0HmB8BhwENtHIu6\n3Pbt21m9+sFR9zv00MOYPn16BSOSJO3tWg0yDwB/HRH/FXgU2NJczMwz93Rg6j6rVz/I+Zd+jVlz\nDxh2n00bnmbZebBo0ZEVjkyStLdqNci8Crij/Nr7xuxFZs09gNnzXjnRw5AkCRhHkImIZcCfZ+bm\nzDyhg2OSJEkak/HcEO+/APs2b4iIW8p7yUiSJFVuPEFmyhDbjqe4j4wkSVLlWl0jU5mImA5cBryD\n4pO2r83MD5e1BcDVwHEUH5twbmbe1nTsyeWxC4G7gCWZ+USV45ckSZ3T6mctVekK4CTg94B3Aksi\nYklZ+zqwBjgS+AqwPCLmA0TE/sBy4BrgKGA9cFO1Q5ckSZ003iCza4zb2iIi5gBnAu/NzHsz8x+A\nS4BjIuIEik/hPisLn6KYdWlc+r0EWJWZl2fmw8AZwIKIOL5T45UkSdUa76mlKyKi+Z4xvcCyiNjU\nvFMb7yOzGHg+M+9seu5lABHxIeC+zNzatP+dFKeZAI4BVjQdtyUi7ivrK+giOwd2kPnIiPt4kzlJ\nknY3niCzgt3vGfOPwMvKP52wEHgyIt4N/BkwHfgy8AlgP4rTSs3WAfPLr0erd43Nz6/lmlvWMGvl\nz4ase5M5SZKGNuYgk5mv7+A4hvNiipvvvQ84nSKcfAF4AZhJsfi32TaKWSLGUB+znp7hz8CNVBuP\n0W4019MzlWnTJnZJ01jf656MtfEa7eqrRmfPq2fPq2fPq1dVr7v9qqUdwCzgHZn5LEBE/CbwfuDv\ngbmD9u+lCDkAW9k9tPQCG8c7iL6+4a8wH6nWTn19M5gzZ9/Rd+zwGMa6356Otaq+6pfsefXsefXs\n+eTT7UFmLbC1EWJKSXF66Dng0EH7zyuPoawPPhU2D7h/vIPo79/CwMDOYWtV6O/fwsaNmyt5rZHG\nMNb9Wh1rT89U+vpmjNhztZc9r549r549r16j553W7UFmJbBPRByYmf9abjuE4p4xK4EPRURvZjZO\nIS3ml58BtbJ8DEBEzAQWAReNdxADAzvZsWPob/yqfiBGGkNVxvpe2zHWbni/ext7Xj17Xj17Pvl0\ndZDJzB9ExC3AdRHxfoo1MhcAH6dYfPxMWbsYOAU4mmItDcC1wNKIOB+4mSLAPJaZ3632XUiSpE6p\nw6qndwH/SjHTch1wRWZ+PjN3UoSXecA9FDfLO7VxGioznwJOo7ivzN3AbOAtlY9ekiR1TFfPyABk\n5iaKWZbTh6g9Dgz7SdyZeStwUKfGJkmSJlYdZmQkSZKGZJCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1\nZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCR\nJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1NW2iB6DR\n7RzYQeYjo+536KGHMX369ApGJElSdzDI1MDm59dyzS1rmLXyZ8Pus2nD0yw7DxYtOrLCkUmSNLEM\nMjUxa+4BzJ73yokehiRJXcU1MpIkqbZqNSMTEbcA6zLzzPLxAuBq4DjgSeDczLytaf+TgcuAhcBd\nwJLMfKLiYUuSpA6pzYxMRLwdeOOgzTcBa4Ajga8AyyNifrn//sBy4BrgKGB9ub8kSZokahFkImIO\nsAy4u2nbiRQzLWdl4VMUsy5nlrssAVZl5uWZ+TBwBrAgIo6vdvSSJKlTahFkgEuA64GHm7YdA9yX\nmVubtt1JcZqpUV/RKGTmFuC+prokSaq5rl8jU868vBY4DLiqqbQfxWmlZuuA+WOsa5Dt27ezevWD\nw9bHci8bSZKq1NVBJiJ6KcLL+zNzW0Q0l2cC2wYdsg3oHWN9zHp6hp+4GqlWtZ6eqUyb1vp4/vmf\nV3P+pV9j1twDhqyve3wVL194dEfH0ehnN/V1srPn1bPn1bPn1auq110dZICPUaxz+fYQta3ASwdt\n6wVeaKoPDi29wMbxDqKvb0ZLtar19c1gzpx99+j4ke5Xs2nDM5WMo/EcqpY9r549r549n3y6Pci8\nDXh5RGwqH/cCRMQfAX8BHDJo/3nA2vLr58rHg+v3j3cQ/f1bGBjYOWytW/T3b2Hjxs17dPxEj6On\nZyp9fTNG7Lnay55Xz55Xz55Xr9HzTuv2IPM64EVNj5cBu4DzgQXAf4uI3sxsnEJaDNxRfr2yfAxA\nRMwEFgEXjXcQAwM72bFj6G/8bvqBGGmcYz2+G8bRrufQ+Njz6tnz6tnzyaerg0xm/sq5jHJmZldm\nPhERTwHPANdFxMXAKcDRwOnl7tcCSyPifOBmigDzWGZ+t6rxS5KkzqrtqqfM3Am8meJ00T3AO4FT\nM/PZsv4UcBrFfWXuBmYDb5mY0UqSpE7o6hmZwTLzjEGPHwdOGGH/W4GDOj0uSZI0MWo7IyNJkmSQ\nkSRJtWWQkSRJtWWQkSRJtWWQkSRJtWWQkSRJtWWQkSRJtWWQkSRJtWWQkSRJtVWrO/uqddu3b2f1\n6gdH3CfzkYpGI0lSexhk9hKrVz/I+Zd+jVlzDxh2n3WPr+LlC4+ucFSSJO0Zg8xeZNbcA5g975XD\n1jdteGbYmiRJ3cggM0nsHNgx4qkhTxtJkiYjg8wksfn5tVxzyxpmrfzZkHVPG0mSJiODzCQy0qkj\nTxtJkiYjL7+WJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCRJEm1ZZCR\nJEm1ZZCRJEm1ZZCRJEm11fWftRQRvw5cAZwAvAD8H+BDmbk9IhYAVwPHAU8C52bmbU3HngxcBiwE\n7gKWZOYTlb4BSZLUMXWYkfkbYB/gNcDbgTcBF5e1rwNrgCOBrwDLI2I+QETsDywHrgGOAtYDN1U6\nckmS1FFdHWQiIoDfBU7PzEcy8x+BC4F3RsQJwCuAs7LwKYpZlzPLw5cAqzLz8sx8GDgDWBARx1f/\nTiRJUid0dZABfgi8ITPXD9r+EuBY4L7M3Nq0/U6K00wAxwArGoXM3ALc11SXJEk119VrZDLzp0Dz\nmpcpwNnA7cB+FKeVmq0D5pdfj1aXJEk11+0zMoP9JbAI+DAwE9g2qL4N6C2/Hq0uSZJqrqtnZJpF\nxKeBDwJ/nJkPRcRW4KWDduuluLIJYCu7h5ZeYON4X7unZ/i8N1Jtb9XTM5Vp01rrS6Of9rU69rx6\n9rx69rx6VfW6FkEmIj4HnAW8KzMbVx49BxwyaNd5wNqm+rwh6veP9/X7+ma0VNtb9fXNYM6cfff4\nOVQte149e149ez75dH2QiYiLgPcBb8vM5U2llcAFEdGbmY1TSIuBO5rqi5ueZybFaamLxjuG/v4t\nDAzsHLamX9Xfv4WNGze3dGxPz1T6+maM2HO1lz2vnj2vnj2vXqPnndbVQSYiDgY+AvwF8L2IeHlT\n+bvAM8B1EXExcApwNHB6Wb8WWBoR5wM3UwSYxzLzu+Mdx8DATnbsGPob3x+I3Y3UryqfQ+Njz6tn\nz6tnzyefbj9ZeArFGD9CcQXSGopTR2sycydwKsXponuAdwKnZuazAJn5FHAaxX1l7gZmA2+p+g1I\nkqTO6eqn++swAAAJp0lEQVQZmcz8NPDpEeqPUXx0wXD1W4GDOjA0SZLUBbo6yHSDzZs3s27dumGn\nIn/yk59UPCJJktRgkBnFf77g4/zrj180bP1HT69m7sLXVDgiSZLUYJAZxfR99mXW/MOHrff/bOuw\nNUmS1FndvthXkiRpWAYZSZJUWwYZSZJUWwYZSZJUWwYZSZJUWwYZSZJUWwYZSZJUWwYZSZJUWwYZ\nSZJUWwYZSZJUWwYZSZJUWwYZSZJUWwYZSZJUWwYZSZJUWwYZSZJUW9MmegCaXHYO7CDzkRH3OfTQ\nw5g+fXpFI5IkTWYGGbXV5ufXcs0ta5i18mdD1jdteJpl58GiRUdWPDJJ0mRkkFHbzZp7ALPnvXKi\nhyFJ2gu4RkaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNWWQUaSJNXWpL/8OiJ6gSuB04AXgM9k\n5qUTO6q912g3zOvpmcprX3tshSOSJNXZpA8ywCXAEcDrgQXA9RHxZGZ+bSIHtbcayw3zru6bwYEH\nHlLxyCRJdTSpg0xEzATeA/x+Zj4APBARy4CzAYPMBBnphnk7B3bw0EMP0d+/hYGBnUPu8/Of/xyA\nF73oRcO+hh+DIEl7h0kdZIDDKd7jXU3b7gT+bGKGo9Fsfn4tl92whllzfzzsPuseX8XMl7ycWXMP\nGLL+0x8/wZI3PULEQSO+1khhZ/v27axe/eCo4zUwSdLEmuxBZj9gfWbuaNq2DtgnIuZm5oYJGpdG\nMNpHHGza8Ayz5u4/7D6bNjzDNbc8NOzpKxg97GQ+UjzHMGFpLM8xlpmjPZ1d6pbA1S3jkLT3mexB\nZiawbdC2xuPedr3Ipg1PD1t74ac/BHaNePxo++xNz9Gu15j5kpeP+BxbN63n8utvZWbf/UPWf7I2\n+TcH/PYeP8c++85hZt+vDfsco+3zQv+POO/0N3DQQQcPWX/kkYe59Lpvjfgaoz3H1KlTePGL9+Fn\nP9vKzp0j93447RjH3qQdPdf4TLaeH3FE93/wbk9PNRdGT9m1q/5/ocOJiD8CrsjMX2/adhCwGpib\nmc9P2OAkSdIem+z3kXkOeFlENL/PecAWQ4wkSfU32YPMPwE/B5pvTPJaYNXEDEeSJLXTpD61BBAR\nfwW8BjgTmA9cB/yHzPz6RI5LkiTtucm+2BfgPIo7+34H+CnwUUOMJEmTw6SfkZEkSZPXZF8jI0mS\nJjGDjCRJqi2DjCRJqi2DjCRJqi2DjCRJqq294fLrcYuIXopLtk8DXgA+k5mXTuyo6i0ifh24AjiB\noqf/B/hQZm6PiAXA1cBxwJPAuZl5W9OxJwOXAQspPsl8SWY+UekbqLmIuAVYl5lnlo8XYM87IiKm\nU/TuHRSf7XZtZn64rC3AvrddRMwH/go4HtgAfDYzP1vWFmDP26b8/+M9wAcyc0W5bQF70OOIOAdY\nCswCbgTOzsytYx2TMzJDuwQ4Ang98H7goog4bUJHVH9/A+xDcXPCtwNvAi4ua18H1gBHAl8Blpe/\nmIiI/YHlwDXAUcB64KZKR15zEfF24I2DNt+EPe+UK4CTgN8D3gksiYglZc3v9c64EdhE8Xv7HOAT\nEfHmsmbP26QMMTcAhwwqtfz7JCLeClwILAFOpLgT/7LxjMv7yAwSETMpGv37mXlHue3DwEmZeeKE\nDq6mIiKAh4CXZ+b6ctvbgb8E/j3FL5pfayTwiLgNuCMzPx4RHwcWN3ofETOAHwJvavxrQMOLiDnA\nAxS/ZB7KzDMj4kSKXyT2vM3Kfq8DTszMO8tt5wOvAv4av9fbLiJmAz8BfiszHyq3/V+K7/nl2PO2\niIiDgf9VPvxt4ITMXLGnv08i4rvAtzPz4rL+GuDvKT7YeUyzMs7I7O5wilNudzVtuxM4ZmKGMyn8\nEHhDI8Q0eQlF+r5v0DfsnRRTlFD0/Re/UDJzC3BfU10juwS4Hni4adsx2PNOWQw83wgxAJm5LDPf\ni9/rnbIF2AycERHTyn84vQa4H3veTq8DbqfozZSm7S3/Pik/0Plo4I6mY1cC0yn+XzwmBpnd7Qes\nz8wdTdvWAftExNwJGlOtZeZPB50vnQKcTfFDsR/Fv5yaraP4XCzGUNcwyn8pvZZfnsJrsOedsxB4\nMiLeHREPR8RjEfGR8nvevndAZm6j+H3ypxSh5mHg7zLzy9jztsnMqzJz6RCzJHvS49kUSw5+Uc/M\nAYp1TmP+O3Cx7+5mUizQa9Z43FvxWCarvwQWUSTx8xi6341eD/f34d/FCMpz2VcB78/MbcU/Un9h\ntJ7a89a9mOI00vuA0yl+iX+BYoG7fe+cg4FvUMxAHgZ8LiJux55XYU96PLPp8XDHj8ogs7ut7N7A\nxuMXKh7LpBMRnwY+CPxxZj4UEVuBlw7arZdf9nq4v4+NHR1o/X0MWJWZ3x6iZs87ZwfFlRfvyMxn\nASLiNykuGvh7YPCsrn3fQxFxEvAeYH45O3N/udD0IxSzvva8s/bk98nWpsfDHT8qTy3t7jngZeW5\nu4Z5wJbMfH6CxjQpRMTngHOBd2VmY9X6cxT9bTYPWDvGuob2NuDUiNgUEZuAdwF/EhH9wLPY805Z\nC2xthJhSUkyT+73eGUcAj5YhpuF+4ADseRX2pMcbKMLML+oR0UMRPsf8d2CQ2d0/AT+nWCTW8Fpg\n1cQMZ3KIiIsoptvflpk3NpVWAkeUp0IaFpfbG/XFTc8zk+K01Eo0ktdRTLEfXv75BsXVG4cD38ee\nd8pKivV0BzZtO4Ti3horgSPte9utAQ6MiOYzDAcDT2DPq9Dq7/C7MnMXxf9bFzcd+2pgO8XVlmPi\n5ddDiIi/olj1fibFv6SuA/5DZn59IsdVV+Vle/8M/AXFjQab/ZjiG/ZfKBalngJ8CDg0M58tp+Uf\nAv4cuBm4CHhlZh5R0fAnhYj4MrCrvPx6Kva8YyLiGxRT7e+nWCNzPfBxihu2/TPwIPa9bSKij2KB\n723AJ4CDgGspenst9rztImIn8Pry8ulWfp+8KjMXlc/1Nor1fKdThNJrKS7HPnes43FGZmjnAfcC\n3wE+B3zUELNHTqH4XvsIxTfqGoppwzWZuRM4lWJq8R6KG4id2piaz8ynKO6wfCZwN8Uq97dU/QYm\nk7Lnb8aed8q7gH+luKT0OuCKzPx82fdTsO9tlZn9FDcg3I+ib58BPp6ZX7LnHfOLGZAWf5+c2nT8\nV4FPUiyKv5Xi1icXjGcwzshIkqTackZGkiTVlkFGkiTVlkFGkiTVlkFGkiTVlkFGkiTVlkFGkiTV\nlkFGkiTVlkFGkiTVlkFGkiTVlkFGkiTVlkFGkiTV1v8HgFR+NaWTigIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111e80978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "messages['length'].plot(bins=50, kind='hist') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the x-axis goes all the way to 1000, we must have a really long message in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    5572.000000\n",
       "mean       80.489950\n",
       "std        59.942907\n",
       "min         2.000000\n",
       "25%        36.000000\n",
       "50%        62.000000\n",
       "75%       122.000000\n",
       "max       910.000000\n",
       "Name: length, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages.length.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we do have a long message with 910 character length! Let's take a look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"For me the love should start with attraction.i should feel that I need her every time around me.she should be the first thing which comes in my thoughts.I would start the day and end it with her.she should be there every time I dream.love will be then when my every breath has her name.my life should happen around her.my life will be named to her.I would cry for her.will give all my happiness and take all her sorrows.I will be ready to fight with anyone for her.I will be in love when I will be doing the craziest things for her.love will be when I don't have to proove anyone that my girl is the most beautiful lady on the whole planet.I will always be singing praises for her.love will be when I start up making chicken curry and end up makiing sambar.life will be the most beautiful then.will get every morning and thank god for the day because she is with me.I would like to say a lot..will tell later..\""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages[messages['length'] == 910]['message'].iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful.\n",
    "\n",
    "Going back to the task at hand, let's try and see typicla length of a ham or spam message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([<matplotlib.axes._subplots.AxesSubplot object at 0x114f8eeb8>,\n",
       "       <matplotlib.axes._subplots.AxesSubplot object at 0x118258128>], dtype=object)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABAMAAAF7CAYAAACn90IlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xu8XGV58P1fskMCgWyhaUtoIY1UeoEUNSIFFLEqvq1P\nW0W0VelJ0EhrrVUeHqgnsB6qxgOoLUUpyOtrX2ut4rHKo7YFrURQKA9GuETOEMCSBhMgB7N3nj/W\njAw7+zQz995zWL/v58OHPetea+baV2b2utc197rvBbt27UKSJEmSJNXHwl4HIEmSJEmS5pfFAEmS\nJEmSasZigCRJkiRJNWMxQJIkSZKkmrEYIEmSJElSzVgMkCRJkiSpZiwGSJIkSZJUMxYDJEmSJEmq\nGYsBkiRJkiTVjMUAaYhExG0RcXGv45AkSZLU3ywGSMNlV68DkCRJktT/LAZIkiRJklQzC3bt8otE\naVhExK3AN4ENwB8By4D/AF6VmTc39nkFcBpwGFVBMIF3ZOY/N9r/GLgAOAE4FzgCuB04A/gBcD5w\nbOM13pSZn5yv30+SJPVWRDwZWAs8haof8W2q/sC3I+KjwCrgH4A3Az/TaD89M/9Py3McD7wB+DVg\nb+Bu4P/NzL9qtP8ScCvwu8AfAM8BHgQ+BHwQOA94IbANuCQz/3Juf2tpODkyQBo+LwUOpyoG/CnV\nyfoTABHxZ1QX+p8B/gdwMtWJ9B8i4hdanmMP4P8H/g74HeBhqhP7F4AvAr9NVQy4ZMJxkiRpSEXE\nMuArwI+AFwAvprqY/0qjDeBJwNuoigG/D/ws8O8RsX/jOZ4AfK3xHL9H1ae4AjgnIn5vwkteCPyf\nxj5fbTzvVcBDjdf/Z+DMiHjhXPy+0rBb1OsAJBV3F/D8zBwDiIhDgDdGxD7AY4F3Z+Y7mztHxO3A\nd4HjgH9qbF4IvD0zP9rY513APwLvz8zzGtt+DHyHqtjw+fn4xSRJUk89nuri/oOZuQ4gIm4EXkk1\nGhFgFPitzPxWo/0q4BbgL6hGAzwBuCwz/6j5pBHxNeD5wK/zSF8E4MuZeU5jn+9TFRfuzczXNNr/\nLSL+AHga8Oniv6005CwGSMPn281CQMOtjf/vm5lnAETEY4BDgccBz6SaeHBJyzG7gCtbHt/X+P9V\nLds2Np+3UNySJKm/fQ/4L+BLEfFPwGXA/87M1wNEBMCtzUIAQGbeGxHfAp7RePxx4OMRsQT4FeAQ\nqtEEi3h0XwRa+iKZ+aPG8181YZ9N2BeROmIxQBo+D014PN74/8KI+GXgw8CzgO3AjcB1jfYFE47b\nPOHxrkmeW5Ik1URmPhQRxwFvohri/0pgW0R8jOqbf6ju/5/oR8BqgIjYE/gbqrkAFlF9afEt4CfM\n3BcB+yJSMRYDpPpYCHyJao6AI4HrMnM8Ig6jml9AkiRpWpl5E/DHEbGAagLAP6Sao+jmxi4/O8lh\n+1MVBKCaAPAk4EXA1zNzK0BE3DfJcZLmkMUAqT5+jmo43l9k5rUt2/8H1bf+TigqSZKm1Jio7++A\nX83MH1GtFPDtiDgZWNnY7VciIjIzG8f8AvBU4K8b7U8D/i0zv9jyvEdS9VPsi0jzyGKAVB/3AbcB\nr46Iu6nusXsujwzr23uG4ycO3ZMkSfXyH1QX7J9rTC68GXgJ1aSBnwZe1mj/QkS8CRgDzgHup1oW\nEKp7/n83Ik4DbqCaL+CNVLc1ztQXkVSQ1TdpuOxq/DeV51Pdy/dR4JNUw/t+m2rugKfP4rlns02S\nJA2hzLwX+A3gAeDvqZYbfhJwUmZe3tjtduC9wLmNfW4EnpaZDzTaTwcupVom8AvAqY2fLwSObdx+\nAFP3OyZun6nvI2kKC3bt6uyz05gB9DvAn2XmFY1tB1FNTvYMqguON2bmp1qOOYHqD8PBVLODrsnM\nW1vaXwucQbU0yaeAV2fmto4ClCRJtTEX/RJJ7YmIjwLPyMyDex2LpJl1NDKgccL9BNVao81tI8C/\nUE1O9iSqiuDHI+LxjfaDqKqAF1GtS34/8NmW418InA2soZrp/BhgbSfxSZKk+piLfokkScOu7WJA\nY+bxdcBjJzT9FvCLwB9m5k2Z+RGqmcuf2mh/BXB1Zp6XmTcApwCrIuL4RvtrgHMz88uZ+V3gNODl\njeVHJEmSdjOH/RJJnXHIvjQgOhkZ8Azg68CxPHpCsWdQLQ/y07U/M/OkzPz7xsNjgCta2rYC11Dd\nG7QQOAr4RsvzrQMWA0/sIEZJklQPxfslcx6xNKQy85TM/OVexyFpdtpeTSAzL2j+HBGtTQcDt0bE\nO6nWG/0v4C2Z+blG+wHAhglPdx9wILAvsGdre2aORcTGRvu3241TkiQNvznql0iSNPRKriawD9UQ\nu32pZif//4B/jognN9qXAtsnHLMdWNJoY5p2SZKkdnTTL5Ekaei1PTJgGjuB+zPzTxuP/zMing68\nEvgTqgl8Jp5gl1Ctdb6t5fHE9odnG8CuXbt2LVjgUuiSpJ7wBNRfuumXzIr9DklSj3V1EipZDLgH\nGJ+wLYEjGj/fDayY0L4CuBbYSHVSXgH8AH46C/DyxvPOyoIFC9i8eStjYxPDUDtGRhYyOrqXuSzE\nfJZjLssyn+U0c6m+0k2/ZFbsd5Tj36NyzGVZ5rMcc1lWib5HyWLAOuCNEbEgM5uziB4G3NbSflxz\n54hYCqwGzs7MXRFxdaO9OZnPU4EdwHXtBDE2Ns7Onb65SjCXZZnPcsxlWeZTQ6rTfsk57byIn5+y\nzGc55rIs81mOuewfJYsBnwDeDJwfEe8FfgP4TeDXGu0XA2dExJnAF6lOtrdkZvPi/3zggohYTzWh\nz/nARzJzG5IkSe3ppF9yc2Ze3otgJUmab91OIPjTdUQzcwvwHKqq+/XAnwO/l5nXNdpvB04CTgWu\noprQ58SW4z8JvBP4MHAZcCVwVpfxSZKk+ui2X/KC+Q5YkqReWbBr166Z9xocuzZteshhJ11atGgh\n++23N+ayDPNZjrksy3yW08ilM8nVj/2OQvx7VI65LMt8lmMuyyrR9yi5tKAkSZIkSRoAFgMkSZIk\nSaoZiwGSJEmSJNWMxQBJkiRJkmrGYoAkSZIkSTVjMUCSJEmSpJqxGCBJkiRJUs1YDJAkSZIkqWYs\nBkiSJEmSVDMWAyRJkiRJqhmLAZIkSZIk1YzFAEmSJEmSasZigCRJkiRJNWMxQJIkSZKkmrEYIEmS\nJElSzVgMkCRJkiSpZiwGSJIkSZJUMxYDJEmSJEmqGYsBkiRJkiTVjMUASZIkSZJqxmKAJEmSJEk1\ns6jXAcyXHTt2sH799dPuc/jhR7B48eJ5ikiSJEmSpN6oTTFg/frrOfP9n2HZ8pWTtm/ZeAdrT4fV\nq4+c58gkSZIkSZpftSkGACxbvpJ9VxzS6zAkSZIkDYmZRiA7+lj9qlbFAEmSJEkqaboRyI4+Vj/r\nuBgQEUuA7wB/lplXTGgbBb4PvCEzP9ay/QTgXOBg4EpgTWbe2tL+WuAMYBnwKeDVmbmt0xglSVI9\nzEW/RJJmyxHIGkQdrSbQOOF+Anj8FLusBQ6YcMxBwKXARcBTgPuBz7a0vxA4G1gDPAs4pvE8kiRJ\nU5qLfokkScOu7WJARBwGrAMeO0X7cVQX8/dOaHoFcHVmnpeZNwCnAKsi4vhG+2uAczPzy5n5XeA0\n4OURsWe7MUqSpHqYw36JJElDrZORAc8Avg4cCyxobYiIxcBHgFcBOyYcdwzw02F7mbkVuAY4NiIW\nAkcB32jZfx2wGHhiBzFKkqR6KN4vmctgJUnqF23PGZCZFzR/joiJzW8EvpuZX5uk7QBgw4Rt9wEH\nAvsCe7a2Z+ZYRGxstH+73TglSdLwm6N+iSRJQ6/YagIR8XjglcARU+yyFNg+Ydt2YEmjjWnaJUmS\nZq3LfokkSUOv5NKCHwHOzsz7p2jfxu4n2CXApkYbU7Q/3E4QIyOT3/kw1faJ+yxa1NGcikOlmavZ\n5EwzM5/lmMuyzGc55rAvddMvmTX/7cvw71E55rKs2eRzplx7jVHxvVlWiTwWKQZExErgqcATIuL9\njc1LgQ9HxIsz87eAu4EVEw5dAVwLbKQ6Ka8AftB4zhFgOXBPO7GMju7V1vaJ++y3397tvNxQm03O\nNHvmsxxzWZb51LAp0C+ZNT8/ZZnPcsxlWdPlc6Zce43xaL43+0epkQF3AY+bsO1y4APAPzQerwOO\nazZGxFJgNVXVfldEXN1ob07m81SqyX6uayeQzZu3MjY2Pun22Ry7adND7bzcUBoZWcjo6F5T5lLt\nMZ/lmMuyzGc5zVyqb3TTLzmnnRfy81OGf4/KMZdlzSafM11neI1R8b1ZVom+R5FiQGaOA7e0bouI\nncCPMrP5zf7FwBkRcSbwRaqT7S2Z2bz4Px+4ICLWU03ocz7wkczcRhvGxsbZuXP3N9ds3nBTHVtX\n5qMs81mOuSzLfGrYdNEvuTkzL2/ntfz8lGU+yzGXZU2Xz5muM/y3eDTz0T+6vdFg12zbMvN24CTg\nVOAqqhUETmxp/yTwTuDDwGXAlcBZXcYnSZLqo9t+yQvmLjRJkvpLVyMDMnNkmraDJ9l2GXDoNMes\nBdZ2E5MkSaqn0v0SSZKGmVM5SpIkSZJUMxYDJEmSJEmqGYsBkiRJkiTVjMUASZIkSZJqxmKAJEmS\nJEk1YzFAkiRJkqSasRggSZIkSVLNWAyQJEmSJKlmLAZIkiRJklQzFgMkSZIkSaoZiwGSJEmSJNWM\nxQBJkiRJkmrGYoAkSZIkSTVjMUCSJEmSpJqxGCBJkiRJUs1YDJAkSZIkqWYsBkiSJEmSVDMWAyRJ\nkiRJqhmLAZIkSZIk1YzFAEmSJEmSasZigCRJkiRJNWMxQJIkSZKkmrEYIEmSJElSzVgMkCRJkiSp\nZiwGSJIkSZJUMxYDJEmSJEmqmUWdHhgRS4DvAH+WmVc0th0DvA94AnAX8N7MvKjlmBOAc4GDgSuB\nNZl5a0v7a4EzgGXAp4BXZ+a2TmOUJEn1MBf9EkmShllHIwMaJ9xPAI9v2bY/8C/AvwJPAt4CfCgi\nnttoXwlcClwEPAW4H/hsy/EvBM4G1gDPAo4B1nYSnyRJqo+56JdIkjTs2i4GRMRhwDrgsROaTgTu\nycw3Z+bNmflJ4GPAyY32VwBXZ+Z5mXkDcAqwKiKOb7S/Bjg3M7+cmd8FTgNeHhF7tv9rSZKkOpjD\nfokkSUOtk5EBzwC+DhwLLGjZ/mWqE+lEj2n8/2jgiubGzNwKXAMcGxELgaOAb7Qctw5YDDyxgxgl\nSVI9FO+XzE2YkiT1l7bnDMjMC5o/R0Tr9juAO1rafh54CdXQf4ADgA0Tnu4+4EBgX2DP1vbMHIuI\njY32b7cbpyRJGn5z1C+RJGnodTyB4HQaQ/s/TXWS/Uhj81Jg+4RdtwNLGm1M0z5rIyOTD3aYavvE\nfRYtcoGFZq5mkzPNzHyWYy7LMp/lmMP+1kG/ZNb8ty/Dv0flmMuyZpPPmXLtNUbF92ZZJfJYvBgQ\nEXsDnwceBzytZTWAbex+gl0CbGq0MUX7w+28/ujoXm1tn7jPfvvt3c7LDbXZ5EyzZz7LMZdlmU8N\nsw77JbPm56cs81mOuSxrunzOlGuvMR7N92b/KFoMiIhlwFeoluh5Zmbe0tJ8N7BiwiErgGuBjVQn\n5RXADxrPNQIsB+5pJ4bNm7cyNjY+6fbZHLtp00PtvNxQGhlZyOjoXlPmUu0xn+WYy7LMZznNXKq/\ndNEvmTU/P2X496gcc1nWbPI503WG1xgV35tlleh7FCsGRMQCqiV6VgHHZ+ZNE3ZZBxzXsv9SYDVw\ndmbuioirG+3NyXyeCuwArmsnjrGxcXbu3P3NNZs33FTH1pX5KMt8lmMuyzKfGkZd9EvOaed1/PyU\nZT7LMZdlTZfPma4z/Ld4NPPRP0qODHgF8OvA7wCbG+v7AuzIzE3AxcAZEXEm8EWqk+0tmdm8+D8f\nuCAi1lPd03c+8JGW4XySJEmz1Um/5ObMvLwXwUqSNN+6nXVgV+M/gJOolvT5ItXFfPO/TwNk5u2N\nfU4FrqJaQeDE5hM11v99J/Bh4DLgSuCsLuOTJEn10W2/5AXzHK8kST3T1ciAzBxp+fm5s9j/MuDQ\nadrXAmu7iUmSJNVT6X6JJEnDzHUdJEmSJEmqGYsBkiRJkiTVjMUASZIkSZJqxmKAJEmSJEk1YzFA\nkiRJkqSasRggSZIkSVLNWAyQJEmSJKlmLAZIkiRJklQzFgMkSZIkSaoZiwGSJEmSJNWMxQBJkiRJ\nkmrGYoAkSZIkSTVjMUCSJEmSpJqxGCBJkiRJUs1YDJAkSZIkqWYsBkiSJEmSVDMWAyRJkiRJqhmL\nAZIkSZIk1cyiXgcgSZIkSf1sx44dXHfddZO2Zd44z9FIZVgMkCRJkqRpfO9713Pm+z/DsuUrd2u7\n75ar2f/go3oQldQdiwGSJEmSNINly1ey74pDdtu+ZeOdPYhG6p5zBkiSJEmSVDMWAyRJkiRJqhmL\nAZIkSZIk1YzFAEmSJEmSaqbjCQQjYgnwHeDPMvOKxrZVwIXAscBtwOsy86stx5wAnAscDFwJrMnM\nW1vaXwucASwDPgW8OjO3dRqjJEmqh7nol0iSNMw6GhnQOOF+Anj8hKbPAhuAI4GPA5dGxIGNYw4C\nLgUuAp4C3N/Yv/mcLwTOBtYAzwKOAdZ2Ep8kSaqPueiXSJI07NouBkTEYcA64LETtj+LqrJ+Wlbe\nRVVlP7Wxyxrg6sw8LzNvAE4BVkXE8Y321wDnZuaXM/O7wGnAyyNiz05+MUmSNPzmsF8iSdJQ62Rk\nwDOAr1MNuVvQsv1o4JoJw/q/2div2X5FsyEztwLXAMdGxELgKOAbLceuAxYDT+wgRkmSVA/F+yVz\nGq0kSX2i7TkDMvOC5s8R0dp0ANVQvFb3AQfOon1fYM/W9swci4iNjfZvtxunJEkafnPUL5EkaeiV\nXE1gKbB9wrbtwJJZtC9teTzV8ZIkSbPVTb9EkqSh1/FqApPYBvzMhG1LgIdb2ieeYJcAmxptTNH+\nMG0YGZm8vjHV9on7LFrkaovNXM0mZ5qZ+SzHXJZlPssxh32pm37JrPlvX4Z/j8oxl2U187hw4YIZ\n9pz+ObzG8L1ZWok8liwG3M3us/iuAO5paV8xSfu1wEaqk/IK4AcAETECLG85flZGR/dqa/vEffbb\nb+92Xm6ozSZnmj3zWY65LMt8akh10y+ZNT8/ZZnPcsxlWfvs0/mc5l5jPJrvzf5RshiwDjgrIpZk\nZnPY3XE8MingusZjACJiKbAaODszd0XE1Y325mQ+TwV2ANe1E8TmzVsZGxufdPtsjt206aF2Xm4o\njYwsZHR0rylzqfaYz3LMZVnms5xmLtVXOu2XnNPOi/j5KcO/R+WYy7Ka+XzwwW0z7zwFrzEqvjfL\nKtH3KFkMuBy4E7gkIt4GPI9qhYCXNdovBs6IiDOBL1KdbG/JzObF//nABRGxnmpCn/OBj0yYBXhG\nY2Pj7Ny5+5trpjfc+NhOvv/970+53+GHH8HixYvbCWXgTZVLdcZ8lmMuyzKfGlKd9EtuzszL23kR\nPz9lmc9yzGVZ4+O7Oj7Wf4tHMx/9o9tiwE8/FZk5HhHPBy4CvgP8EDgxM+9qtN8eEScBHwDOBv4D\nOLHl+E9GxC8BH6ZaUvCfgbO6jG/WHnrgHi760gaWrXtwt7YtG+9g7emwevWR8xWOJElqX7f9khfM\nf8iSJPVGV8WAzByZ8PgW4JnT7H8ZcOg07WuBtd3E1I1ly1ey74pDevXykiSpC6X7JZIkDTOncpQk\nSZIkqWYsBkiSJEmSVDMWAyRJkiRJqhmLAZIkSZIk1YzFAEmSJEmSasZigCRJkiRJNWMxQJIkSZKk\nmrEYIEmSJElSzVgMkCRJkiSpZiwGSJIkSZJUMxYDJEmSJEmqGYsBkiRJkiTVjMUASZIkSZJqxmKA\nJEmSJEk1YzFAkiRJkqSasRggSZIkSVLNWAyQJEmSJKlmLAZIkiRJklQzFgMkSZIkSaoZiwGSJEmS\nJNWMxQBJkiRJkmrGYoAkSZIkSTVjMUCSJEmSpJqxGCBJkiRJUs1YDJAkSZIkqWYsBkiSJEmSVDOL\nSj5ZRBwI/B1wPLAR+EBmfqDRtgq4EDgWuA14XWZ+teXYE4BzgYOBK4E1mXlryfgkSVK9dNM3kSRp\nmJUeGfApYAvwZOC1wDsi4vmNts8BG4AjgY8DlzZO0ETEQcClwEXAU4D7gc8Wjk2SJNVPR30TSZKG\nXbFiQETsCxwNvD0zb87MzwNfAZ4dEc8EHguclpV3UX37f2rj8DXA1Zl5XmbeAJwCrIqI40vFJ0mS\n6qXLvokkSUOt5MiArcBDwCkRsSgiAngacC1wDHBNZm5r2f+bVMPyoDpRX9FsyMytwDUt7ZIkSe3q\npm8iSdJQK1YMyMztwKuBP6E6+d4A/EtmfhQ4gGoYXqv7gOZQvJnaJUmS2tJl30SSpKFWdAJB4DDg\n88B7gSOAD0XE14GlwPYJ+24HljR+nql91kZGJq9vTLW9neddtKgeiy80c9VtzlQxn+WYy7LMZznm\nsK912jeZFf/ty/DvUTnmsqxmHhcuXNDVc9TlOmI6vjfLKpHHYsWAiHg28HLgwEYl/trGJDxvAr4O\nLJ9wyBLg4cbP29j95LsE2NRuHKOje7W1vZ3n3W+/vbt6jkHTbc70aOazHHNZlvnUsOqybzIrfn7K\nMp/lmMuy9tlnz46PreN1xHR8b/aPkiMDngzc1DjZNl0LvAG4Gzh8wv4rgHsaP9/deDyx/dp2g9i8\neStjY+OTbu/G5s1b2bTpoa6eY1CMjCxkdHSvKXOp9pjPcsxlWeaznGYu1Xe66ZvMip+fMvx7VI65\nLKuZzwcf3DbzzlOo03XEdHxvllWi71GyGLABeFxELMrMnY1thwG3AuuA10fEkpYT8nHANxo/r2s8\nBiAilgKrgXPaDWJsbJydO3d/c3X7hpvqeYdZHX/nuWQ+yzGXZZlPDbFu+iaz4uenLPNZjrksa3x8\nV8fH+m/xaOajf5QsBnwBWAv8fUS8AzgUeH3jvyuAO4FLIuJtwPOAo4CXNY69GDgjIs4EvkhVBLg5\nMy8vGJ8kSaqXbvomkiQNtZKrCWwGnk01O+9VwPuAt2bm32fmONVJdgXwHeBk4MTMvKtx7O3ASVRr\n+14F7Au8oFRskiSpfrrpm0iSNOyKriaQmTcCvzFF2y3AM6c59jKqir0kSVIR3fRNJEkaZq7rIEmS\nJElSzVgMkCRJkiSpZiwGSJIkSZJUMxYDJEmSJEmqGYsBkiRJkiTVjMUASZIkSZJqxmKAJEmSJEk1\nYzFAkiRJkqSasRggSZIkSVLNWAyQJEmSJKlmFvU6AEmSJEkaRuNjO8m8ccr2ww8/gsWLF89jRNIj\nLAZIkiRJ0hx46IF7uOhLG1i27sHd2rZsvIO1p8Pq1Uf2IDLJYoAkSZIkzZlly1ey74pDeh2GtBvn\nDJAkSZIkqWYsBkiSJEmSVDMWAyRJkiRJqhmLAZIkSZIk1YzFAEmSJEmSasZigCRJkiRJNWMxQJIk\nSZKkmrEYIEmSJElSzVgMkCRJkiSpZiwGSJIkSZJUMxYDJEmSJEmqGYsBkiRJkiTVzKKSTxYRi4Fz\ngZcC24GLM/ONjbZVwIXAscBtwOsy86stx57QOPZg4EpgTWbeWjI+SZJUL930TSRJGmalRwZ8EHg2\n8BzgZGBNRKxptH0O2AAcCXwcuDQiDgSIiIOAS4GLgKcA9wOfLRybJEmqn476JpIkDbtiIwMiYj/g\nVOBZmfndxrb3AkdHxA+BxwJHZ+Y24F0R8ezG/m8F1gBXZ+Z5jeNOAe6NiOMz84pSMUqSpProsm8i\nSdJQKzky4Djggcz8ZnNDZq7NzFcAxwDXNE62Td+kGpYHcDRwRctxW4FrWtolSZLa1U3fRJKkoVZy\nzoCDgdsi4g+BNwCLgY8C7wAOoBqG1+o+oDkUb6Z2SZKkdnXTN5EkaaiVLAbsA/wK8ErgZVQn2Q8D\nDwNLqSbtabUdWNL4eaZ2SZKkdnXTN5EkaaiVLAbsBJYBL83MuwAi4peAVwH/G1g+Yf8lVCdjgG3s\nfvJdAmxqN4iRkcnvfJhqezvPu2hRPVZibOaq25ypYj7LMZdlmc9yzGHf6qZvMiv+25fh36NyzGVZ\nzTwuXLhgzp7fawx1okQeSxYD7gG2NU+2DUk13O5u4PAJ+69oHEOjfcUk7de2G8To6F5tbW/neffb\nb++unmPQdJszPZr5LMdclmU+NcS66ZvMip+fssxnOeayrH322XNOntdrDPVSyWLAOmDPiHhcZv6w\nse3xVOv2rgNeHxFLMrM5JO844Bstxx7XfKKIWAqsBs5pN4jNm7cyNjY+6fZOjY/t5Kqrrpn2OX71\nV49g8eLFHb9GPxkZWcjo6F5T5lLtMZ/lmMuyzGc5zVyq73TTN5kVPz9l+PeoHHM5vR07dvC9710/\nZfvEPn0znw8+uG3KY7qxefNWNm16aE6eu9/43iyrRN+jWDEgM38QEV8CLomIV1Hdl3cW1fI8VwB3\nNtreBjwPOIrq/j2Ai4EzIuJM4ItURYCbM/PyduMYGxtn587d31zdvOEeeuAeLvzCBpZ9a8uk7Vs2\n3sHa08dZvfrIjl+jH02VS3XGfJZjLssynxpWXfZNZsXPT1nmsxxzObnrrruOM9//GZYtX7lb23R9\n+vHxXXMSTx3/ner4O/erkiMDAH4f+BBVVf1h4IOZ+bcAEfE84CLgO8APgRObw/Yy8/aIOAn4AHA2\n8B/ACwrH1pVly1ey74pDeh2GJElqT0d9E0nDy369VClaDMjMLVQV9ZdN0nYL8Mxpjr0MOLRkPJIk\nqd666ZtIkjTMnMpRkiRJkqSasRggSZIkSVLNWAyQJEmSJKlmLAZIkiRJklQzFgMkSZIkSaoZiwGS\nJEmSJNWMxQBJkiRJkmrGYoAkSZIkSTVjMUCSJEmSpJqxGCBJkiRJUs1YDJAkSZIkqWYsBkiSJEmS\nVDMWAyRJkiRJqplFvQ5AkiRJknptfGwnmTc+atvIyEJGR/fixhtv6FFU0tyxGCBJkiSp9h564B4u\n+tIGlq2jocSRAAAZxElEQVR7cLe2+265mv0PPqoHUUlzx2KAJEmSJAHLlq9k3xWH7LZ9y8Y7exCN\nNLcsBkiSJEkaGjt27GD9+usnbZt4G4BUZxYDJEmSJA2N9euv58z3f4Zly1fu1uZwf+kRFgMkSZIk\nDRWH+0szc2lBSZIkSZJqxmKAJEmSJEk1YzFAkiRJkqSacc4ASZIkSQPFFQOk7lkMkCRJkjRQXDFA\n6p7FAEmSJEkDxxUDpO44Z4AkSZIkSTUzZyMDIuJLwH2ZeWrj8SrgQuBY4DbgdZn51Zb9TwDOBQ4G\nrgTWZOatcxVfSeNjO6e9N+nww49g8eLF8xiRJElq1W6/RJKkYTcnxYCIeAnwXOCSls2fBa4DjgRe\nAFwaEYdm5l0RcRBwKfBm4DLgnMb+T5yL+Ep76IF7uOhLG1i27sHd2rZsvIO1p8Pq1Uf2IDJJktRu\nv2T+I5Qkaf4VLwZExH7AWuCqlm3PovrG/5jM3Aa8KyKeDZwKvBVYA1ydmec19j8FuDcijs/MK0rH\nOBemumdJkiT1Tof9EkmSht5czBnwXuBjwA0t244GrmmccJu+STU0r9n+04v+zNwKXNPSLkmS1IlO\n+iWSJA29osWARqX96cDbJjQdAGyYsO0+4MBZtkuSJLWli36JJElDr9htAhGxBLgAeFVmbo+I1ual\nwPYJh2wHlsyyfdZGRiavb0y1fT6MjCxk0aLBWbihmate5myYmM9yzGVZ5rMcc9h/uuyXzJr/9mX4\n96icuuRyWH6/QbtO6EZd3pvzpUQeS84Z8Baq+/6/NknbNuBnJmxbAjzc0j7xBLwE2NRuEKOje7W1\nfT6Mju7Ffvvt3bPX71QvczaMzGc55rIs86kh9RY675fMmp+fssxnOcOey2H5/Qb1OqEbw/JvNwxK\nFgNeDOwfEVsaj5cARMSLgL8GHj9h/xXAPY2f7248nth+bbtBbN68lbGx8Um398rmzVvZtOmhnr1+\nu0ZGFjI6uteUuVR7zGc55rIs81lOM5fqK930S2bNz08Z/j0qpy657GXfvqRBu07oRl3em/OlRN+j\nZDHgGcAeLY/XAruAM4FVwF9GxJLMbA7LOw74RuPndY3HAETEUmA11RKDbRkbG2fnzt3fXL18w00V\nU78b1Lj7lfksx1yWZT41pLrpl8yan5+yzGc5w57LYbmYHPZ/p8nU8XfuV8WKAZl5Z+vjRiV+V2be\nGhG3A3cCl0TE24DnAUcBL2vsfjFwRkScCXyRqghwc2ZeXio+SZJUH132SyRJGnrzMntDZo4Dz6ca\ngvcd4GTgxMy8q9F+O3AS1fq+VwH7Ai+Yj9gkSVK9zNQvkSSpDkreJvAomXnKhMe3AM+cZv/LgEPn\nKh5JklRf7fZLJEkadq7rIEmSJElSzVgMkCRJkiSpZiwGSJIkSZJUMxYDJEmSJEmqGYsBkiRJkiTV\njMUASZIkSZJqxmKAJEmSJEk1YzFAkiRJkqSasRggSZIkSVLNWAyQJEmSJKlmLAZIkiRJklQzFgMk\nSZIkSaoZiwGSJEmSJNWMxQBJkiRJkmrGYoAkSZIkSTWzqNcBqHM7duxg/frrp93n8MOPYPHixfMU\nkSRJkiRpEFgMGGDr11/Pme//DMuWr5y0fcvGO1h7OqxefeQ8RyZJkiRJ6mcWAwbcsuUr2XfFIb0O\nQ5IkSZI0QCwGSJIkSeo7090Sm3njPEcjDR+LAZIkSZL6znS3xN53y9Xsf/BRPYhKGh4WA+bY+NjO\naSuXP/nJTwDYY489ptzHSQAlSZJUR1PdErtl4509iEYaLhYD5thDD9zDRV/awLJ1D07aft8tV7P0\nMfs7CaAkSZIkad5YDJgH003yt2XjnSxbfpCTAEqSJEmS5s3CXgcgSZIkSZLml8UASZIkSZJqpuht\nAhHxC8AHgWcCDwP/BLw+M3dExCrgQuBY4DbgdZn51ZZjTwDOBQ4GrgTWZOatJeOTJEn10k3fRJKk\nYVZ6ZMCngT2BpwEvAX4HeFuj7XPABuBI4OPApRFxIEBEHARcClwEPAW4H/hs4dgkSVL9dNQ3kSRp\n2BUbGRARAfwasH9m3t/Ydjbwnoj4CvBY4OjM3Aa8KyKeDZwKvBVYA1ydmec1jjsFuDcijs/MK0rF\nOIimW5pwuiULJUmquy77JpIkDbWStwncC/xm82Tb4jHAMcA1jZNt0zephuUBHA389KI/M7dGxDWN\n9loXA6ZbmvC+W65m/4OP6kFUkiQNhG76JpIkDbVixYDM/DHQOgfAAuDVwNeBA6iG4bW6D2gOxZup\nvdamWppwy8Y7exCNJEmDocu+iaQ27Nixg/Xrr5+y/fDDj2Dx4sXzGJGkmRSdQHCC9wCrgaOA04Ht\nE9q3A0saPy+doV2SJKlb7fRNJLVh/frrOfP9n2HZ8pW7tW3ZeAdrT4fVq4/sQWSSpjInxYCIeDfw\nGuD3MvP7EbEN+JkJuy2hmtUXYBu7n3yXAJvafe2RkcnnRJxq+7AbGVnIokXt/e7NXNU1Z6WZz3LM\nZVnmsxxz2P866JvMiv/2Zfj3qJxe5XJkZOGUo1mb7Z32SYdZJ3kZVH7OyyqRx+LFgIj4EHAa8PuZ\n2VwR4G7g8RN2XQHc09K+YpL2a9t9/dHRvdraPuxGR/div/327vhYlWM+yzGXZZlPDbsO+yaz4uen\nLPNZznzncqbX66RPWof3Qzd99UFVh3/XQVG0GBAR5wCvBF6cmZe2NK0DzoqIJZnZHJJ3HPCNlvbj\nWp5nKdUwvnPajWHz5q2MjY1Pur2ONm/eyqZND7V1zMjIQkZH95oyl2qP+SzHXJZlPstp5lL9p4u+\nyaz4+SnDv0fl9CqXM/W1O+mT1qH/3kleBpWf87JK9D1KLi14GPAm4K+Bb0XE/i3NlwN3ApdExNuA\n51Hdr/eyRvvFwBkRcSbwRaoiwM2ZeXm7cYyNjbNz5+5vrrq+4abKx1wfq92Zz3LMZVnmU8Oqy77J\nrPj5Kct8ljPfuZypr91JPHXov9fxPV/H37lflbxh43mN53sT1ey8G6iG2m3IzHHgRKrhd98BTgZO\nzMy7ADLzduAkqrV9rwL2BV5QMDZJklQ/HfdNJEkadiWXFnw38O5p2m8GnjlN+2XAoaXikSRJ9dZt\n30SSpGHmVI6SJEmSJNXMnCwt2CvvOe/DfC/vZNeuXbu13XfvnbD3E3oQlSRJkiRJ/WWoigE33XYP\n/730qEnbfrxoCXvMczySJEmSJPWjoSoGSJIkSZrejh07WL/++inbDz/8CBYvXlzs9cbHdpJ547y9\nnqTZsRggSZIk1cj69ddz5vs/w7LlK3dr27LxDtaeDqtXH1ns9R564B4u+tIGlq17cF5eT9LsWAyQ\nJEmSambZ8pXsu+KQoX09STNzNQFJkiRJkmrGkQGSJEnSAJrs3v+RkYWMju7F5s1bOfTQw/v+Xvzp\n5hOYbp4BSd2zGCBJkiQNoJnv/R/v+3vxp5tP4L5brmb/gydfKUxS9ywGSJIkSQNqGO7Fn+p32LLx\nzh5EI9WHcwZIkiRJklQzjgyQJEmSNKPJ5iho8v5+afBYDJAkSZI0o+nmKPD+fmnwWAyQJEmSNCve\n3y8ND4sBkiRJUovphsMDHH74EX2/ZF+nXOpv/kyXaxju95n6g8UASZIkqcXMS/bR90v2dcql/ubP\ndLke9veZ+oPFAEmSJM2Lqb5xHxlZyOjoXqxc+TgWLpyf7ulMk+F1smTfsIwo8FaA+TMMS0NqcFkM\nkCRJ0ryY6Rv39/2vF/GEJ6zueSydfgNe5xEFkgaPxQBJkiTNm376JnQuvgHvp99PkqazsNcBSJIk\nSZKk+eXIAEmSJKlPzTS3wVRcFUDSTCwGDDGXK5EkSRpsnc5t4KoAkmZiMWCIuVyJJElS/+t0ZYOZ\n5jZwVQBJ07EYMOScxEaSJKm/zcXKBpI0E4sBkiRJNTXdN9Iw+LcUdnq/fS/4Lb5aTXe7709+8hMA\n9thjjymPH/TPruaHxQBJkqSamu4b6WG4pdBv3DWoZprzYelj9p/0fQ3D8dnV/OirYkBELAHOB04C\nHgbel5nv721UkiRpGNnvqHRyS+EgjSjol2/cnd1f7Zruvbts+UEd3Qo8SJ9dzb2+KgYA7wWeDPw6\nsAr4WETclpmf6WVQw2i6E9LIyEJGR/di5crHsXBhv71FJEkqxn5Hh4Z9RMFccHZ/9QM/u2rVN1d6\nEbEUeDnwG5l5HXBdRKwFXg14Ui5suhMSwI//61ZOe/4TOOSQmLR9pnuVrCpKkvqZ/Y7uTfWtZaff\ngI+P7eTGG29gbGx80vZ+6Vt08w1/v4xSUL05wbia+qYYADyRKp4rW7Z9E3hDb8IZftP9Idiy8U4u\n/MJ6li3fMmn7dPcq/fi/bmXN79xIxKFTvvZUJ/SZhi5Nd6wkSW0YuH7HdOfIuSjSd3rR2+k34A89\ncA8XfmHDpH2P6foW0/3uczH83m/41e/m+5aUQbn1YFDinE/9VAw4ALg/M3e2bLsP2DMilmfmxh7F\nVVszFQumuldpy8Y7uehL35921MFUJ/TMG6tjp5gQpdPOwGzaof+KFBZHJGnO9GW/48477+Av33ou\n+zxm+W5t9971Qx7e4xennAxvqiJ9p0N/u7no7fQb8OmOm6pvMd3vPlcX537Dr3423wWrQbn1YFDi\nnE/9VAxYCmyfsK35eEmJF9iy8Y4p2x7+8b3Arrbb5vLYQY5r6WP2n/LYbVvu57yPXcbS0Wt3a/vv\ne5KfW/mEjo/dc+/9WDr685MeO1P7w5t/xOkv+00OPfSw3dpuvPEG3n/JVzo6duHCBeyzz548+OA2\nxsenzulkunndYdRNLrW7Qc7nk5/cXyfrkZGFvQ5B7SvS7yj9b79160NsW/rLLPnZx+/Wtm3jOOyc\nGPLs3HRTThrrTTfllP2jmc7n0x3XSf+jm77FdOY7Ttts62Vbs72Tz+6WjXdw003L2v67dtNNOWP7\nyMjCnvc7ZopzZGQhixYNzvm8xPlnwa5d/dEBjIgXAR/MzF9o2XYosB5YnpkP9Cw4SZI0VOx3SJLq\nrp9KH3cDPxsRrTGtALZ6QpYkSYXZ75Ak1Vo/FQP+E/gJcEzLtqcDV/cmHEmSNMTsd0iSaq1vbhMA\niIi/A54GnAocCFwC/HFmfq6XcUmSpOFjv0OSVGf9NIEgwOnA+cC/Aj8G3uwJWZIkzRH7HZKk2uqr\nkQGSJEmSJGnu9dOcAZIkSZIkaR5YDJAkSZIkqWYsBkiSJEmSVDMWAyRJkiRJqhmLAZIkSZIk1Uy/\nLS3YlohYDiwBHs7MB3odjyRJGm72PSRJw2LglhaMiJOAVwNHA3u2NG0FrgbOc41gSZJUin0PSdIw\nGqhiQEScDpwDrAW+CdwHbKeq0K8Ang78T+DNmfmhXsU5aCLieOBY4EAa33YA9wDrMvPyXsY2aMxl\nWeazHHMpdca+R1kRcSBwKlP8PQIuysy7ehehpFLse5QzV7kctGLA3cCrpqu+R8SJwIcy86D5i2ww\nRcRjgc8Cq4Br2L2Dsxq4GXhBZt7eozAHgrksy3yWYy7nTkTsCTyR3U/M12Xmtl7GpnLse5QTEc8B\nLgWuZPLCynHAUcCJmflvvYpzEHnRVY657J59j3LmOpeDNmfAUuC2Gfa5C3jM3IcyFC4EbgCOycyt\nExsjYinwUeAjwG/Mc2yDxlyWZT7LMZeFNYoAa4GXA4uBjTxyYl4O7IiIjwBnZeaOngWqUux7lHMu\n8PbMfNdUO0TEXwIfAJ4wb1ENsGkuFEaBJwFviAgvumbBXBZl36OcOc3loBUDPgNcEhGvAa7MzJ3N\nhohYCBwDXAB8ukfxDZpjgadM9sYCyMyHI+KvgKvmN6yBZC7LMp/lmMvy/obqfPP/UH1TNNZsiIgR\n4KnA+Y39XtmTCFWSfY9yVlGNDJjO54Gz5z6UoeFFVznmshz7HuXMaS4HrRjwKuC9wGXAooi4n0e+\njflZ4CfAx4DTexbhYLmF6o/ZDdPs89tU33hoeuayLPNZjrks73eBZ2Xmdyc2NAoD34iIU6nOVRYD\nBp99j3KupPp29bTJbqWJiCXAm4Fvz3tkg8uLrnLMZTn2PcqZ01wOVDEgM7cDfx4RZ1Hdp3kA1fC9\nbcDdwH9O9QHWpF4HXBoRzwOuADaw+717TwNO6lmEg8NclmU+yzGX5W0Bfn6GfX4B8BaBIWDfo6g1\nwOeAH0XENez+9+hI4E7geT2LcPB40VWOuSzHvkc5c5rLgZpAUOVFxEFUJ+ej2b2Dsw642PuiZsdc\nlmU+yzGXZUXEa6mGMZ8LfIPJT8z/C3hnZq7tVZxSv4qIZ/Hov0fbgXupPk9XZOZ4D8MbKBFxAtWt\nF1czw4VCZl7WqzgHgbksy75HOY1cvoLqtrSiubQYIElSmyLihcBrqGY+b113fhtVR/L8zPxkL2KT\n+lVE/BPwiszc3Hi8B/Aeqttp9gTuB96dme/rXZSDx4uucubyokvqRxYDas71fssxl2WZz3LM5dxp\nTCC3L490GDdmpidWaRIRMQYckJk/ajx+J3Ay1TDYG6iWyFoLXJCZb+9ZoJK6Zt+jjIh4FVURalvL\ntucDfwr8InAj8J7M7GguC4sBNeZ6v+WYy7LMZznmcu5Msxb1lZl5RS9jk/pRRIwDK1qKAT8E/mdm\nfq5ln+cCF2bmgT0Kc+B40VXGXF901Yl9j3ImKaL+EdWKFh/hkSLqHwAvbf1bOlsDNYGginO933LM\nZVnmsxxzWdgs16K+haqT43BS6RG7Gv81jQG3TtjnZmDZvEU04GZx0fV04PSI8KJrZh8C/plqlNfE\ni67PUV10/XtEdHTRVTP2PcpZMOHx6cAZmfk3zQ0RcS3w11Tv07ZYDKi3VbjebymrMJclrcJ8lrIK\nc1maa1FLnVkAXBgR3wN+AHwX+Avg5QARsSfV36J1PYtw8HjRVc6cXnTVzCrse5QycRj/cuDfJ2y7\njGoJ3LYt7OQgDY3mer97Ttboer9tMZdlmc9yzGV5xwJ/Nd1a1MBfUc06LekRJ1Gt0f5LVEWA5wEv\ni4h9G+13AccDr+1NeANpFbO76Hrc3Icy8GZ70bVqPoIZcPY9ylkA/HFEnBARK4EvAydM2OdE4KZO\nntyRAfU203q/T6Za7/fEnkU4ONZQDRueKZfP71mEg8W1qMsxl+W5FrXUgcz8LNW58qciYmVmPtB4\neDLwrcx8cN6DG1zNi67TWu91b/Kiqy3Ni65rqUauNC+6vteyT8cXXTXjNUY5HwKeQ7WC0S9SFa3G\nI+KSzHwgIr5KVUR9USdP7gSCIiKeydRLqFzuer+zN8nayduoLgi+jbls2yTvza088t50Leo2mMty\nXItaUr+IiFVUBZaDqeYwmfLLiMy8pUdhDoSI+ABwWOO/n150AT838aIrM7/Qu0gHh9cYZUXEMqr3\n56GZ+bHGtr8CvpCZ3+nkOS0G1FhELAbeRlWJfwzwdeCNmfn9ln32BzZk5khvohx8EbEZeGJmTpwk\nSdOIiDXA0Zn5iohYQLX01GlUMyXfRrWO+9/2MMSB0ZgN+dnAtZn50Yg4GXgTsJJq8q4PZObf9zLG\nQeO63pL6yRRfRnjR1aG5uOiqE68xypkkl18D3lQqlxYDaiwi3gf8Do9M3vFqqplSf78xlK/55ron\nM51fYhoRcfE0zX9ANVRqC0BmnjovQQ2wiHgH1YXWezNzbUS8Cfhzqkl7Eng8cBbwwcx8R+8i7X8R\n8RfAO4CvUH1b/S9UQ8neDVxL1dl5PfDWzPxQr+KUJEnDwWuMcibkcgFVLp9EoVw6Z0C9/R7wksz8\nD4CI+CTwHuCfIuL3M/NTjf2sGM1sf+C5VBMjTXYf8QJ2n6VWUzsVeHHLMkinAKc1/+gBX4mI9cAl\nVBe6mtpfACdn5ucjIqjeny9rftMBfDkibgLeR3VfmmbBdb0l9YOIOH62+2bmFXMZy6Azl0V5jVHO\nxFz+IwVzaTGg3pYCG5sPMnMXcEZEjAH/EBE7gW/1KrhBkpm/FREvAdZSDd95W2ZuB4iIFwFneq9e\nW5YAm1se76C6yGp1D9V7WNNbDqxv/Hwz1bre10/Y50bg5+YzqEHmut6S+sjfUo2Wg+m/dNgFOBx7\neuayHK8xypnTXFoMqLd/A94bES/LzPubGzPzrMY62f8ITLlurR4tM/8xIprrfF4fEa/KzK/1Oq4B\n9QmqP3CvyMxvUt0e8N6IeGlm3hURjwPOZ+bllFRNcPe2iHg71QiLbVQnkVMzc3tELALeSDWqRbPj\nut6S+sVTqM6ZjwWOnWxFAc2auSzHa4xy5jSX3qNRb6+h+tbwvsY3XT+Vmc37s9/Qi8AGVWZuysyX\nA38CnB8R/4Cfs068jmpt33+NiB9RzRdwBHB7RDxENW/Afze2a3p/StWx+R7V+/LVVDNL3xUR36Ka\nefo5VLcTaHZW4brekvpAYxTiSxsP397LWAaduSzKa4xy5jSXTiAoGvcR35uZP56k7TDgeZn57vmP\nbLA11vZ9C/Bi4BmZeWdvIxo8EbEf1TJtBwP7ADtp3JOdmT/oZWyDJiL2Bba23L7ybKolpzZQzY68\nebrj9YjG8lIbqOaxmGpd70uAFZn5zHkOT1INNfprz8jMC3ody6Azl+V4jVHOXOXSYoAkSW1wXW9J\nkjQMLAZIktQB1/WWJEmDzGKAJEmSJEk142oCkiS1wbWoJUnSMLAYIElSe1yLWpIkDTyLAZIktce1\nqCVJ0sBz/XNJktrgWtSSJGkYWAyQJKlNjYLAycAPex2LJElSJ1xNQJIkSZKkmnFkgCRJkiRJNWMx\nQJIkSZKkmrEYIEmSJElSzVgMkCRJkiSpZiwGSJIkSZJUMxYDJEmSJEmqGYsBkiRJkiTVzP8FBYTx\ntTxcZ/IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1124850b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "messages.hist(column='length', by='label', bins=50,figsize=(12,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very interesting! It seems that spam messages tend to have more characters.\n",
    "\n",
    "Now let's begin to process the data so we can eventually use it with SciKit Learn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data is not yet ready to be passed into a classification algorithm, because it's all text. We need it to be in the form of a numerical feature vector to make it suitable for classification. \n",
    "\n",
    "The simplest way to accomplish this is to convert it into a bag-of-words format, where each word in the text is represented by a number.\n",
    "\n",
    "As a first step, we'll write a function thats splits a message into its individual words and return a list. We'll also remove very common words or stopwords, ('the', 'a', etc..) using the NlTK library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "mess = 'Sample message! Notice: it has punctuation.'\n",
    "\n",
    "# Check characters to see if they are in punctuation\n",
    "nopunc = [char for char in mess if char not in string.punctuation]\n",
    "\n",
    "# Join the characters again to form the string.\n",
    "nopunc = ''.join(nopunc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing a list of stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stopwords.words('english')[0:10] # Show some stop words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sample', 'message', 'Notice', 'it', 'has', 'punctuation']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nopunc.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now just remove any stopwords\n",
    "clean_mess = [word for word in nopunc.split() if word.lower() not in stopwords.words('english')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sample', 'message', 'Notice', 'punctuation']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_mess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's put both of these together in a function to apply it to the whole DataFrame later on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def text_process(mess):\n",
    "    \n",
    "    # Check characters to see if they are in punctuation\n",
    "    nopunc = [char for char in mess if char not in string.punctuation]\n",
    "\n",
    "    # Join the characters again to form the string.\n",
    "    nopunc = ''.join(nopunc)\n",
    "    \n",
    "    # Removing stopwords and returning the list\n",
    "    return [word for word in nopunc.split() if word.lower() not in stopwords.words('english')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the original DataFrame again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>message</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "      <td>61</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                            message  length\n",
       "0   ham  Go until jurong point, crazy.. Available only ...     111\n",
       "1   ham                      Ok lar... Joking wif u oni...      29\n",
       "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...     155\n",
       "3   ham  U dun say so early hor... U c already then say...      49\n",
       "4   ham  Nah I don't think he goes to usf, he lives aro...      61"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's \"tokenize\" these messages. Tokenization is just the term used to describe the process of converting the normal text strings in to a list of tokens (words that we actually want).\n",
    "\n",
    "Let's see an example output on on column:\n",
    "\n",
    "**Note:**\n",
    "We may get some warnings or errors for symbols we didn't account for or that weren't in Unicode (like a British pound symbol)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [Go, jurong, point, crazy, Available, bugis, n...\n",
       "1                       [Ok, lar, Joking, wif, u, oni]\n",
       "2    [Free, entry, 2, wkly, comp, win, FA, Cup, fin...\n",
       "3        [U, dun, say, early, hor, U, c, already, say]\n",
       "4    [Nah, dont, think, goes, usf, lives, around, t...\n",
       "Name: message, dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check to make sure its working\n",
    "messages['message'].head(5).apply(text_process)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>message</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "      <td>61</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                            message  length\n",
       "0   ham  Go until jurong point, crazy.. Available only ...     111\n",
       "1   ham                      Ok lar... Joking wif u oni...      29\n",
       "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...     155\n",
       "3   ham  U dun say so early hor... U c already then say...      49\n",
       "4   ham  Nah I don't think he goes to usf, he lives aro...      61"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show original dataframe\n",
    "messages.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continuing Normalization\n",
    "\n",
    "There are a lot of ways to continue normalizing this text. Such as [Stemming](https://en.wikipedia.org/wiki/Stemming) or distinguishing by [part of speech](http://www.nltk.org/book/ch05.html).\n",
    "\n",
    "NLTK has lots of built-in tools and great documentation on a lot of these methods. Sometimes they don't work well for text-messages due to the way a lot of people tend to use abbreviations or shorthand, For example:\n",
    "    \n",
    "    'Nah dawg, IDK! Wut time u headin to da club?'\n",
    "    \n",
    "versus\n",
    "\n",
    "    'No dog, I don't know! What time are you heading to the club?'\n",
    "    \n",
    "Some text normalization methods will have trouble with this type of shorthand and so I'll leave you to explore those more advanced methods through the [NLTK book online](http://www.nltk.org/book/).\n",
    "\n",
    "For now we will just focus on using what we have to convert our list of words to an actual vector that SciKit-Learn can use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently, we have the messages as lists of tokens (also known as [lemmas](http://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html)) and now we need to convert each of those messages into a vector the SciKit Learn's algorithm models can work with.\n",
    "\n",
    "Now we'll convert each message, represented as a list of tokens (lemmas) above, into a vector that machine learning models can understand.\n",
    "\n",
    "We'll do that in three steps using the bag-of-words model:\n",
    "\n",
    "1. Count how many times does a word occur in each message (Known as term frequency)\n",
    "\n",
    "2. Weigh the counts, so that frequent tokens get lower weight (inverse document frequency)\n",
    "\n",
    "3. Normalize the vectors to unit length, to abstract from the original text length (L2 norm)\n",
    "\n",
    "Let's begin the first step:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each vector will have as many dimensions as there are unique words in the SMS corpus.  We will first use SciKit Learn's **CountVectorizer**. This model will convert a collection of text documents to a matrix of token counts.\n",
    "\n",
    "We can imagine this as a 2-Dimensional matrix. Where the 1-dimension is the entire vocabulary (1 row per word) and the other dimension are the actual documents, in this case a column per text message. \n",
    "\n",
    "For example:\n",
    "\n",
    "<table border = “1“>\n",
    "<tr>\n",
    "<th></th> <th>Message 1</th> <th>Message 2</th> <th>...</th> <th>Message N</th> \n",
    "</tr>\n",
    "<tr>\n",
    "<td><b>Word 1 Count</b></td><td>0</td><td>1</td><td>...</td><td>0</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td><b>Word 2 Count</b></td><td>0</td><td>0</td><td>...</td><td>0</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td><b>...</b></td> <td>1</td><td>2</td><td>...</td><td>0</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td><b>Word N Count</b></td> <td>0</td><td>1</td><td>...</td><td>1</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "Since there are so many messages, we can expect a lot of zero counts for the presence of that word in that document. Because of this, SciKit Learn will output a [Sparse Matrix](https://en.wikipedia.org/wiki/Sparse_matrix)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of arguments and parameters that can be passed to the CountVectorizer. In this case we will just specify the **analyzer** to be our own previously defined function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11444\n"
     ]
    }
   ],
   "source": [
    "# Might take awhile...\n",
    "bow_transformer = CountVectorizer(analyzer=text_process).fit(messages['message'])\n",
    "\n",
    "# Print total number of vocab words\n",
    "print(len(bow_transformer.vocabulary_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take one text message and get its bag-of-words counts as a vector, putting to use our new `bow_transformer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "U dun say so early hor... U c already then say...\n"
     ]
    }
   ],
   "source": [
    "message4 = messages['message'][3]\n",
    "print(message4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see its vector representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 4073)\t2\n",
      "  (0, 4638)\t1\n",
      "  (0, 5270)\t1\n",
      "  (0, 6214)\t1\n",
      "  (0, 6232)\t1\n",
      "  (0, 7197)\t1\n",
      "  (0, 9570)\t2\n",
      "(1, 11444)\n"
     ]
    }
   ],
   "source": [
    "bow4 = bow_transformer.transform([message4])\n",
    "print(bow4)\n",
    "print(bow4.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that there are seven unique words in message number 4 (after removing common stop words). Two of them appear twice, the rest only once. Let's go ahead and check and confirm which ones appear twice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "U\n",
      "say\n"
     ]
    }
   ],
   "source": [
    "print(bow_transformer.get_feature_names()[4073])\n",
    "print(bow_transformer.get_feature_names()[9570])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use **.transform** on our Bag-of-Words (bow) transformed object and transform the entire DataFrame of messages. Let's go ahead and check out how the bag-of-words counts for the entire SMS corpus is a large, sparse matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "messages_bow = bow_transformer.transform(messages['message'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Sparse Matrix:  (5572, 11444)\n",
      "Amount of Non-Zero occurences:  50795\n"
     ]
    }
   ],
   "source": [
    "print('Shape of Sparse Matrix: ', messages_bow.shape)\n",
    "print('Amount of Non-Zero occurences: ', messages_bow.nnz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sparsity: 0\n"
     ]
    }
   ],
   "source": [
    "sparsity = (100.0 * messages_bow.nnz / (messages_bow.shape[0] * messages_bow.shape[1]))\n",
    "print('sparsity: {}'.format(round(sparsity)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the counting, the term weighting and normalization can be done with [TF-IDF](http://en.wikipedia.org/wiki/Tf%E2%80%93idf), using scikit-learn's `TfidfTransformer`.\n",
    "\n",
    "____\n",
    "### So what is TF-IDF?\n",
    "TF-IDF stands for *term frequency-inverse document frequency*, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query.\n",
    "\n",
    "One of the simplest ranking functions is computed by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model.\n",
    "\n",
    "Typically, the tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document; the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.\n",
    "\n",
    "**TF: Term Frequency**, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization: \n",
    "\n",
    "*TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).*\n",
    "\n",
    "**IDF: Inverse Document Frequency**, which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as \"is\", \"of\", and \"that\", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following: \n",
    "\n",
    "*IDF(t) = log_e(Total number of documents / Number of documents with term t in it).*\n",
    "\n",
    "See below for a simple example.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "Consider a document containing 100 words wherein the word cat appears 3 times. \n",
    "\n",
    "The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.\n",
    "____\n",
    "\n",
    "Let's go ahead and see how we can do this in SciKit Learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 9570)\t0.538562626293\n",
      "  (0, 7197)\t0.438936565338\n",
      "  (0, 6232)\t0.318721689295\n",
      "  (0, 6214)\t0.299537997237\n",
      "  (0, 5270)\t0.297299574059\n",
      "  (0, 4638)\t0.266198019061\n",
      "  (0, 4073)\t0.408325899334\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "tfidf_transformer = TfidfTransformer().fit(messages_bow)\n",
    "tfidf4 = tfidf_transformer.transform(bow4)\n",
    "print(tfidf4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll go ahead and check what is the IDF (inverse document frequency) of the word `\"u\"` and of word `\"university\"`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.28005242674\n",
      "8.5270764989\n"
     ]
    }
   ],
   "source": [
    "print(tfidf_transformer.idf_[bow_transformer.vocabulary_['u']])\n",
    "print(tfidf_transformer.idf_[bow_transformer.vocabulary_['university']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To transform the entire bag-of-words corpus into TF-IDF corpus at once:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5572, 11444)\n"
     ]
    }
   ],
   "source": [
    "messages_tfidf = tfidf_transformer.transform(messages_bow)\n",
    "print(messages_tfidf.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many ways the data can be preprocessed and vectorized. These steps involve feature engineering and building a \"pipeline\". I encourage you to check out SciKit Learn's documentation on dealing with text data as well as the expansive collection of available papers and books on the general topic of NLP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With messages represented as vectors, we can finally train our spam/ham classifier. Now we can actually use almost any sort of classification algorithms. For a [variety of reasons](http://www.inf.ed.ac.uk/teaching/courses/inf2b/learnnotes/inf2b-learn-note07-2up.pdf), the Naive Bayes classifier algorithm is a good choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be using scikit-learn here, choosing the [Naive Bayes](http://en.wikipedia.org/wiki/Naive_Bayes_classifier) classifier to start with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "spam_detect_model = MultinomialNB().fit(messages_tfidf, messages['label'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try classifying our single random message and checking how we do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted: ham\n",
      "expected: ham\n"
     ]
    }
   ],
   "source": [
    "print('predicted:', spam_detect_model.predict(tfidf4)[0])\n",
    "print('expected:', messages.label[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastic! We've developed a model that can attempt to predict spam vs ham classification!\n",
    "\n",
    "## Part 6: Model Evaluation\n",
    "Now we want to determine how well our model will do overall on the entire dataset. Let's begin by getting all the predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ham' 'ham' 'spam' ..., 'ham' 'ham' 'ham']\n"
     ]
    }
   ],
   "source": [
    "all_predictions = spam_detect_model.predict(messages_tfidf)\n",
    "print(all_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use SciKit Learn's built-in classification report, which returns [precision, recall,](https://en.wikipedia.org/wiki/Precision_and_recall) [f1-score](https://en.wikipedia.org/wiki/F1_score), and a column for support (meaning how many cases supported that classification). Check out the links for more detailed info on each of these metrics and the figure below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<img src='https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/700px-Precisionrecall.svg.png' width=400 />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        ham       0.98      1.00      0.99      4825\n",
      "       spam       1.00      0.85      0.92       747\n",
      "\n",
      "avg / total       0.98      0.98      0.98      5572\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print (classification_report(messages['label'], all_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are quite a few possible metrics for evaluating model performance. Which one is the most important depends on the task and the business effects of decisions based off of the model. For example, the cost of mis-predicting \"spam\" as \"ham\" is probably much lower than mis-predicting \"ham\" as \"spam\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above \"evaluation\",we evaluated accuracy on the same data we used for training. **You should never actually evaluate on the same dataset you train on!**\n",
    "\n",
    "Such evaluation tells us nothing about the true predictive power of our model. If we simply remembered each example during training, the accuracy on training data would trivially be 100%, even though we wouldn't be able to classify any new messages.\n",
    "\n",
    "A proper way is to split the data into a training/test set, where the model only ever sees the **training data** during its model fitting and parameter tuning. The **test data** is never used in any way. This is then our final evaluation on test data is representative of true predictive performance.\n",
    "\n",
    "## Train Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4457 1115 5572\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "msg_train, msg_test, label_train, label_test = \\\n",
    "train_test_split(messages['message'], messages['label'], test_size=0.2)\n",
    "\n",
    "print(len(msg_train), len(msg_test), len(msg_train) + len(msg_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test size is 20% of the entire dataset (1115 messages out of total 5572), and the training is the rest (4457 out of 5572). Note the default split would have been 30/70.\n",
    "\n",
    "## Creating a Data Pipeline\n",
    "\n",
    "Let's run our model again and then predict off the test set. We will use SciKit Learn's [pipeline](http://scikit-learn.org/stable/modules/pipeline.html) capabilities to store a pipeline of workflow. This will allow us to set up all the transformations that we will do to the data for future use. Let's see an example of how it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('bow', CountVectorizer(analyzer=text_process)),  # strings to token integer counts\n",
    "    ('tfidf', TfidfTransformer()),  # integer counts to weighted TF-IDF scores\n",
    "    ('classifier', MultinomialNB()),  # train on TF-IDF vectors w/ Naive Bayes classifier\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can directly pass message text data and the pipeline will do our pre-processing for us! We can treat it as a model/estimator API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('bow', CountVectorizer(analyzer=<function text_process at 0x11e795bf8>, binary=False,\n",
       "        decode_error='strict', dtype=<class 'numpy.int64'>,\n",
       "        encoding='utf-8', input='content', lowercase=True, max_df=1.0,\n",
       "        max_features=None, min_df=1, ngram_range=(1, 1), preprocessor=None,...f=False, use_idf=True)), ('classifier', MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.fit(msg_train,label_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "predictions = pipeline.predict(msg_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        ham       1.00      0.96      0.98      1001\n",
      "       spam       0.75      1.00      0.85       114\n",
      "\n",
      "avg / total       0.97      0.97      0.97      1115\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(predictions,label_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a classification report for our model on a true testing set! There is a lot more to Natural Language Processing than what we've covered here, and its vast expanse of topic could fill up several college courses! I encourage you to check out the resources below for more information on NLP!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More Resources\n",
    "\n",
    "Check out the links below for more info on Natural Language Processing:\n",
    "\n",
    "[NLTK Book Online](http://www.nltk.org/book/)\n",
    "\n",
    "[Kaggle Walkthrough](https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words)\n",
    "\n",
    "[SciKit Learn's Tutorial](http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Good Job!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
